Image-based Decision Support Systems: Technical Concepts, Design Knowledge, and Applications for Sustainability

Unstructured data accounts for 80-90% of all data generated, with image data contributing its largest portion. In recent years, the field of computer vision, fueled by deep learning techniques, has made significant advances in exploiting this data to generate value. However, often computer vision models are not sufficient for value creation. In these cases, image-based decision support systems (IB-DSSs), i.e., decision support systems that rely on images and computer vision, can be used to create value by combining human and artificial intelligence. Despite its potential, there is only little work on IB-DSSs so far. In this thesis, we develop technical foundations and design knowledge for IBDSSs and demonstrate the possible positive effect of IB-DSSs on environmental sustainability. The theoretical contributions of this work are based on and evaluated in a series of artifacts in practical use cases: First, we use technical experiments to demonstrate the feasibility of innovative approaches to exploit images for IBDSSs. We show the feasibility of deep-learning-based computer vision and identify future research opportunities based on one of our practical use cases. Building on this, we develop and evaluate a novel approach for combining human and artificial intelligence for value creation from image data. Second, we develop design knowledge that can serve as a blueprint for future IB-DSSs. We perform two design science research studies to formulate generalizable principles for purposeful design — one for IB-DSSs and one for the subclass of image-mining-based decision support systems (IM-DSSs). While IB-DSSs can provide decision support based on single images, IM-DSSs are suitable when large amounts of image data are available and required for decision-making. Third, we demonstrate the viability of applying IBDSSs to enhance environmental sustainability by performing life cycle assessments for two practical use cases — one in which the IB-DSS enables a prolonged product lifetime and one in which the IB-DSS facilitates an improvement of manufacturing processes. We hope this thesis will contribute to expand the use and effectiveness of imagebased decision support systems in practice and will provide directions for future research

An Uncertainty-based Human-in-the-loop System for Industrial Tool Wear Analysis

Convolutional neural networks have shown to achieve superior performance on image segmentation tasks. However, convolutional neural networks, operating as black-box systems, generally do not provide a reliable measure about the confidence of their decisions. This leads to various problems in industrial settings, amongst others, inadequate levels of trust from users in the model's outputs as well as a non-compliance with current policy guidelines (e.g., EU AI Strategy). To address these issues, we use uncertainty measures based on Monte-Carlo dropout in the context of a human-in-the-loop system to increase the system's transparency and performance. In particular, we demonstrate the benefits described above on a real-world multi-class image segmentation task of wear analysis in the machining industry. Following previous work, we show that the quality of a prediction correlates with the model's uncertainty. Additionally, we demonstrate that a multiple linear regression using the model's uncertainties as independent variables significantly explains the quality of a prediction ($R^2=0.718$). Within the uncertainty-based human-in-the-loop system, the multiple regression aims at identifying failed predictions on an image-level. The system utilizes a human expert to label these failed predictions manually. A simulation study demonstrates that the uncertainty-based human-in-the-loop system increases performance for different levels of human involvement in comparison to a random-based human-in-the-loop system. To ensure generalizability, we show that the presented approach achieves similar results on the publicly available Cityscapes dataset.Comment: Alexander Treiss and Jannis Walk contributed equally in shared first authorship. To be published at ECML-PKDD 202

Towards Leveraging End-of-Life Tools as an Asset: Value Co-Creation based on Deep Learning in the Machining Industry

Sustainability is the key concept in the management of products that reached their end-of-life. We propose that end-of-life products have—besides their value as recyclable assets—additional value for producer and consumer. We argue this is especially true for the machining industry, where we illustrate an automatic characterization of worn cutting tools to foster value co-creation between tool manufacturer and tool user (customer) in the future. In the work at hand, we present a deep-learning-based computer vision system for the automatic classification of worn tools regarding flank wear and chipping. The resulting Matthews Correlation Coefficient of 0.878 and 0.644 confirms the feasibility of our system based on the VGG-16 network and Gradient Boosting. Based on these first results we derive a research agenda which addresses the need for a more holistic tool characterization by semantic segmentation and assesses the perceived business impact and usability by different user groups

Artificial Intelligence for Sustainability: Facilitating Sustainable Smart Product-Service Systems with Computer Vision

The usage and impact of deep learning for cleaner production and sustainability purposes remain little explored. This work shows how deep learning can be harnessed to increase sustainability in production and product usage. Specifically, we utilize deep learning-based computer vision to determine the wear states of products. The resulting insights serve as a basis for novel product-service systems with improved integration and result orientation. Moreover, these insights are expected to facilitate product usage improvements and R&D innovations. We demonstrate our approach on two products: machining tools and rotating X-ray anodes. From a technical standpoint, we show that it is possible to recognize the wear state of these products using deep-learning-based computer vision. In particular, we detect wear through microscopic images of the two products. We utilize a U-Net for semantic segmentation to detect wear based on pixel granularity. The resulting mean dice coefficients of 0.631 and 0.603 demonstrate the feasibility of the proposed approach. Consequently, experts can now make better decisions, for example, to improve the machining process parameters. To assess the impact of the proposed approach on environmental sustainability, we perform life cycle assessments that show gains for both products. The results indicate that the emissions of CO2 equivalents are reduced by 12% for machining tools and by 44% for rotating anodes. This work can serve as a guideline and inspire researchers and practitioners to utilize computer vision in similar scenarios to develop sustainable smart product-service systems and enable cleaner production

Design knowledge for deep-learning-enabled image-based decision support systems — evidence from power line maintenance decision-making [in press]

With the ever-increasing societal dependence on electricity, one of the critical tasks in power supply is maintaining the power line infrastructure. In the process of making informed, cost-effective, and timely decisions, maintenance engineers must rely on human-created, heterogeneous, structured, and also largely unstructured information. The maturing research on vision-based power line inspection driven by advancements in deep learning offers first possibilities to move towards more holistic, automated, and safe decision-making. However, (current) research focuses solely on the extraction of information rather than its implementation in decision-making processes. This paper addresses this shortcoming by designing, instantiating, and evaluating a holistic deep-learning-enabled image-based decision support system artifact for power line maintenance at a German distribution system operator in southern Germany. Following the design science research paradigm two main components of the artifact are designed: A deep-learning-based model component responsible for automatic fault detection of power line parts as well as a user-oriented interface responsible for presenting the captured information in a way that enables more informed decisions. As a basis for both components, preliminary design requirements from literature and the application field are derived. Drawing on justificatory knowledge from deep learning as well as decision support systems, tentative design principles are derived. Based on these design principles, a prototype of the artifact is implemented that allows for rigorous evaluation of the design knowledge in multiple evaluation episodes, covering different angles. Through a technical experiment the technical novelty of the artifact\u27s capability to capture selected faults (regarding insulators and safety pins) on unmanned aerial vehicle (UAV)-captured image data (model component) is validated. Subsequent interviews, surveys, and workshops in a natural environment confirm the usefulness of the model as well as the user interface component. The evaluation provides evidence that (1) the image processing approach manages to address the gap of power line component inspection and (2) that the proposed holistic design knowledge for image-based decision support systems enables more informed decision-making. This paper therefore contributes to research and practice in three ways. First, the technical feasibility to detect certain maintenance-intensive parts of power lines with the help of unique UAV image data is shown. Second, the distribution system operators specific problem is solved by supporting decisions in maintenance with the proposed image-based decision support system. Third, precise design knowledge for image-based decision support systems is formulated that can inform future system designs of a similar nature

Designing Chemical Emergency Response Systems Based on Open Data

Emergency situations call for reliable information to allow for an effective and timely response. While digitalization penetrates industry sectors at an increasing rate, advances in information technology are not fully leveraged in emergency situations today. Building upon the paradigm of openness in form of open data, we propose a solution that connects government-provided emergency services with private sector expertise and resources. We apply a design science research approach to design an artifact that provides information in real-time to fire departments and medical staff in case of chemical substance incidents. We showcase that by providing open data, private sector organizations acknowledge their responsibility to share critical data while creating value in the process. Our contribution in this paper is threefold: First, we derive design requirements for artifacts to address chemical substance incidents. Second, we design and evaluate an artifact to showcase its suitability. Third, we showcase value creation through open data

Design Knowledge for Deep-Learning-Enabled Image-Based Decision Support Systems

With the ever-increasing societal dependence on electricity, one of the critical tasks in power supply is maintaining the power line infrastructure. In the process of making informed, cost-effective, and timely decisions, maintenance engineers must rely on human-created, heterogeneous, structured, and also largely unstructured information. The maturing research on vision-based power line inspection driven by advancements in deep learning offers first possibilities to move towards more holistic, automated, and safe decision-making. However, (current) research focuses solely on the extraction of information rather than its implementation in decision-making processes. The paper addresses this shortcoming by designing, instantiating, and evaluating a holistic deep-learning-enabled image-based decision support system artifact for power line maintenance at a German distribution system operator in southern Germany. Following the design science research paradigm, two main components of the artifact are designed: A deep-learning-based model component responsible for automatic fault detection of power line parts as well as a user-oriented interface responsible for presenting the captured information in a way that enables more informed decisions. As a basis for both components, preliminary design requirements are derived from literature and the application field. Drawing on justificatory knowledge from deep learning as well as decision support systems, tentative design principles are derived. Based on these design principles, a prototype of the artifact is implemented that allows for rigorous evaluation of the design knowledge in multiple evaluation episodes, covering different angles. Through a technical experiment the technical novelty of the artifact’s capability to capture selected faults (regarding insulators and safety pins) in unmanned aerial vehicle (UAV)-captured image data (model component) is validated. Subsequent interviews, surveys, and workshops in a natural environment confirm the usefulness of the model as well as the user interface component. The evaluation provides evidence that (1) the image processing approach manages to address the gap of power line component inspection and (2) that the proposed holistic design knowledge for image-based decision support systems enables more informed decision-making. The paper therefore contributes to research and practice in three ways. First, the technical feasibility to detect certain maintenance-intensive parts of power lines with the help of unique UAV image data is shown. Second, the distribution system operators’ specific problem is solved by supporting decisions in maintenance with the proposed image-based decision support system. Third, precise design knowledge for image-based decision support systems is formulated that can inform future system designs of a similar nature

Data-centric Artificial Intelligence

Data-centric artificial intelligence (data-centric AI) represents an emerging paradigm emphasizing that the systematic design and engineering of data is essential for building effective and efficient AI-based systems. The objective of this article is to introduce practitioners and researchers from the field of Information Systems (IS) to data-centric AI. We define relevant terms, provide key characteristics to contrast the data-centric paradigm to the model-centric one, and introduce a framework for data-centric AI. We distinguish data-centric AI from related concepts and discuss its longer-term implications for the IS community